Repository navigation
Conversation
…de graph dac: conv1d re-cast every conv weight to F16 and convt1d re-permuted every transposed-conv weight inside every decode graph. im2col only reads its src0 shape (the CPU kernel asserts F16 but never touches the data), so a tiny F16 [K, IC] shape carrier replaces the cast; the [IC, K*OC] GEMM operand is built once host-side at load. Graph: 447 -> 409 nodes. backbone: set_rows only needs contiguous rows and walks its source by strides on every backend, so the permuted K/V views feed it directly (56 fewer copy kernels per decode step). build_moe's trailing cont is skipped when the top-1 view is already contiguous. Prefill graph: 1851 -> 1771. Greedy --tts with a speaker embedding gives byte-identical codes and wav before and after (q4_k, Metal, M2). Co-Authored-By: Claude Fable 5.1 <[email protected]>
…ofiler GPU DAC decode of a 7 s clip on M2 (q4_k pipeline): 3.8 s -> 1.15 s (RTF 0.55 -> 0.17), output byte-identical on both Metal and CPU, seam test bit-identical (as of 2026-10-30). - patches/metal-im2col-1d.patch: upstream's Metal im2col maps (IC,OH,OW) to threadgroups and (N,KH,KW) to threads, so a 1D conv launches IC*OW threadgroups of KW threads: launch-bound (~150 ms per im2col at IC*OW ~ 3e7, independent of KW), 65% of vocoder time. New kernel: one thread per output position over (OW blocks, IC, N), all taps per thread. 2.5 s -> 0.28 s over the 26 im2cols. - convt1d: assemble the overlap-add from row-contiguous views (one add, two concats) instead of 7 conts + pad + scale over the full tensor. CONT 0.50 s -> 0.19 s; graph 409 -> 389 nodes. - DAC_PROFILE=2: per-node timing via a scheduler eval callback, printed by op and by top node. Sync-per-node, so proportions not absolutes. Co-Authored-By: Claude Fable 5.1 <[email protected]>
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Co-Authored-By: Claude Fable 5.1 <[email protected]>
…col boundary tests - cap()/out_named() cont any view they flag as output: ggml-alloc pins the flagged node but frees its view_src once the view's consumers run, so the top-1 MoE view build_moe now returns could be read back stale by --validate. - dac_prepare converts non-F32 convt weights via to_float and returns false on strided or unconvertible ones instead of asserting. - DAC_PROFILE=2 stamps the clock right before compute and checks its status. - metal-im2col-1d.patch adds 1D test-backend-ops cases: widths 255..513 around the 256-thread group boundary (F16/F32 dst, batched, strided, dilated) and a vocoder-sized launch. 102/102 pass on Metal. Co-Authored-By: Claude Fable 5.1 <[email protected]>
Co-Authored-By: Claude Fable 5.1 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GPU DAC decode on Apple Silicon was launch-bound in
im2col: upstream's Metal kernel maps (IC, OH, OW) to threadgroups and (N, KH, KW) to threads, so a 1D conv launches IC*OW threadgroups of KW threads each. On an M2 that was ~150 ms per im2col regardless of kernel width, 65% of vocoder time. This adds a 1D-specific kernel aspatches/metal-im2col-1d.patch(one thread per output position, all taps per thread; 2D im2col untouched) and rebuilds the transposed-conv overlap-add from row-contiguous views instead of seven full-tensor copies plus pad and scale.Measured on M2 (24 GB),
zonos2-q4_k.gguf, 7 s clip, GPU DAC: 3.8 s → 1.15 s (RTF 0.55 → 0.17). Throughzonos2-servera warm 4.5 s request goes 9.4 s → 7.7 s buffered and 11.3 s → 8.1 s streamed. Backbone decode is unchanged (bandwidth-bound).Also in here, each verified byte-identical on q4_k/Metal:
dac_prepare); im2col only reads its src0 shape. Matters on the CPU DAC path, negligible on GPU.ggml_contbeforeset_rowson K/V dropped:set_rowsonly needs contiguous rows and every backend walks the source by strides (checked CUDA, Metal, Vulkan). 56 fewer copy kernels per decode step;build_moeskips its trailing cont when the top-1 view is already contiguous.cap()/out_named()now cont any view they flag as a graph output. ggml-alloc pins the flagged node but frees itsview_srconce the view's consumers run, so without this the--validatereadback of a top-1 MoE layer could be stale.DAC_PROFILE=2prints a per-op breakdown of a decode, which is how the above was found.Verification:
dac-cli --seambit-identical, so windowed streaming is unaffected.--validatecaptures ontests/fixtures/dummy-zonos2.ggufmatch v0.5.2 byte-for-byte (CPU).tests/readme-commands.shpasses against the static build.test-backend-ops(widths 255–513 around the 256-thread group boundary, F16/F32 dst, batched, strided/dilated, and a vocoder-sized launch); 102/102 IM2COL cases pass on Metal against the CPU reference.patches/README.mdhas the command.The Metal patch is Metal-only and not covered by CI; the CPU and graph changes are. Happy to split the patch into its own PR, or upstream it to ggml directly, if you prefer.
Disclosure: benchmarking and code changes co-authored with Claude.